Project Part 1

By Julian Hautzmayer k11904007

Overview

In this notebook I tested four different minimization techniques:

  • steepest descent
  • newton method
  • quasi newton
  • conjugate gradient (linear and non-linear).

For simplification all used code is extracted to different .py files:

  • main.py - provides the main logic
  • problems - definition of problem functions, gradients and hessians, I used 5 different explicit functions
    • sphere function (modified to the power of 4)
    • matyas function (modified to the power of 4)
    • booth function
    • himmelblau function
    • rosenbrock function
  • methods - definition of the different minimization algorithms and methods to represent the results

pls have a look at the implementation as here you will only see the result and conclusion.

In [2]:
import main   # from this script I trigger all methods with examples
import methods as m

Quadratic objective functions

First I use my implementations of steepest descent, newton method, quasi newton and linear conjugate gradient to solve 5 randomly generated quadratic objective functions.

For every random generated example all four methods are applied. For every applied method we start from a random point and make a number of iterations. Here you can see how in every case the loss decreases and converges. After all iterations you can see which point was reached.

At the end we want to see if our result really found a minimum. Therefore we plug in the steps we took in the gradient and see that again in every case the gradient converges against 0 proving that we indeed found a minimum in every case. After every method run with an example this can be seen in the plot.

In [3]:
main.test_all_qof_methods()

##############Example-0##############
Trying steepest_descent
starting steepest descent from x0=[7. 3. 3. 2. 3. 4. 3. 1. 2. 6.]
0) loss: 8266.149163909395
100) loss: 8.309236791196652
200) loss: 6.4008588925354974
300) loss: 5.558268681117596
400) loss: 4.84973032598793
500) loss: 4.299459329230474
600) loss: 3.900896435597632
700) loss: 3.519338704952466
800) loss: 3.1934554743116594
900) loss: 2.9355652040291353
999) loss: 2.71536911698228
finished after 999 iterations at x=[3.40773804 7.41191295 2.83086831 1.58078704 6.57956215 8.41322271
 7.66922165 1.27089911 4.50233613 4.12751789]
Check convergence

---------------------------------------
Trying newton
starting newton method from x0=[5. 2. 6. 8. 6. 6. 7. 5. 1. 3.]
0) loss: 1351.1006624230483
1) loss: 4.302592486970705e-11
finished after 1 iterations at x=[3. 8. 3. 1. 8. 8. 8. 2. 3. 3.]
Check convergence

---------------------------------------
Trying quasi_newton
starting quasi newton from x0=[8. 2. 6. 7. 6. 3. 5. 7. 3. 4.]
0) loss: 970.0657933581431
1) loss: 333.1468019964294
2) loss: 105.48146237371637
3) loss: 60.75276946873307
4) loss: 55.83970661070261
5) loss: 53.0707068643184
6) loss: 40.31518957691151
7) loss: 23.161812490236343
8) loss: 21.576630857050308
9) loss: 4.20476591610087
10) loss: 0.9034738538959828
11) loss: 0.00032033606532982906
12) loss: 0.00032033606532982906
finished after 12 iterations at x=[2.99999754 8.00000105 3.00000352 1.00000136 8.00000253 7.99999618
 8.00000322 2.00000347 2.99999251 2.99999642]
Check convergence

---------------------------------------
Trying linear_conjugate_gradient
starting linear conjugate gradient from x0=[8. 3. 7. 7. 2. 2. 4. 1. 6. 1.]
0) loss: 651.2565862170464
11) loss: 2.6840939793266015e-08
finished after 11 iterations at x=[3. 8. 3. 1. 8. 8. 8. 2. 3. 3.]
Check convergence

##############Example-1##############
Trying steepest_descent
starting steepest descent from x0=[8. 8. 5. 2. 2. 8. 8. 2. 1. 4.]
0) loss: 9206.475275587287
100) loss: 39.25594754791753
200) loss: 24.927552606036016
300) loss: 16.422355931249843
400) loss: 10.94190042399641
500) loss: 7.244398712316797
600) loss: 4.905236366170566
700) loss: 3.32057685853435
800) loss: 2.2510062306456238
900) loss: 1.505154287529301
999) loss: 1.0289284311402924
finished after 999 iterations at x=[6.73330808 4.5424522  3.80800101 7.6042584  6.69223502 0.94623934
 7.25720749 7.9686269  8.0743351  8.14063877]
Check convergence

---------------------------------------
Trying newton
starting newton method from x0=[5. 6. 2. 8. 8. 3. 4. 7. 2. 2.]
0) loss: 10294.597709478501
1) loss: 4.1091864530094624e-08
finished after 1 iterations at x=[6. 5. 4. 8. 7. 1. 7. 8. 8. 8.]
Check convergence

---------------------------------------
Trying quasi_newton
starting quasi newton from x0=[1. 3. 3. 6. 8. 7. 3. 4. 5. 5.]
0) loss: 486.3927911124359
1) loss: 150.8795429373285
2) loss: 135.34981955197455
3) loss: 117.9696076639739
4) loss: 111.51241778696193
5) loss: 102.19140385579473
6) loss: 91.92001810068993
7) loss: 85.51422570961464
8) loss: 51.29818886782415
9) loss: 1.288737848179713
10) loss: 0.0132975928801215
11) loss: 0.0132975928801215
finished after 11 iterations at x=[3.34197951 7.09865882 4.70310924 9.25545081 8.247603   1.33258898
 5.80149481 7.91382256 7.34816119 7.62123966]
Check convergence

---------------------------------------
Trying linear_conjugate_gradient
starting linear conjugate gradient from x0=[7. 8. 1. 4. 1. 2. 4. 6. 8. 2.]
0) loss: 503.8166943351024
10) loss: 0.003363800811299935
finished after 10 iterations at x=[6.70981805 4.43955837 3.81223597 7.66473382 6.66683062 0.91118401
 7.32005718 8.02301279 8.17407136 8.10114705]
Check convergence

##############Example-2##############
Trying steepest_descent
starting steepest descent from x0=[8. 4. 1. 5. 3. 2. 8. 8. 8. 5.]
0) loss: 8847.195600866977
100) loss: 8.722218640600143
200) loss: 3.7878839858397373
300) loss: 1.900412188070416
400) loss: 0.9491539929889813
500) loss: 0.5052975219069837
600) loss: 0.29516368759926814
700) loss: 0.21109900663039277
800) loss: 0.18680666061330897
900) loss: 0.1770758032795231
999) loss: 0.17964689907597192
finished after 999 iterations at x=[5.98802555 4.68200954 7.47177151 2.34218695 2.20105471 2.09483638
 3.06104638 3.30191111 4.19945196 1.68112195]
Check convergence

---------------------------------------
Trying newton
starting newton method from x0=[6. 3. 7. 5. 3. 3. 1. 4. 6. 5.]
0) loss: 4105.916341086359
1) loss: 1.029533138649498e-09
finished after 1 iterations at x=[5. 6. 7. 3. 3. 1. 2. 3. 4. 3.]
Check convergence

---------------------------------------
Trying quasi_newton
starting quasi newton from x0=[5. 2. 7. 8. 3. 3. 8. 6. 2. 7.]
0) loss: 842.5294637303654
1) loss: 250.97036353516324
2) loss: 144.72158745083456
3) loss: 120.85741870867034
4) loss: 109.41975668914182
5) loss: 106.01616163861894
6) loss: 97.21851830389505
7) loss: 83.91244128029683
8) loss: 66.61391903226044
9) loss: 1.160134421414776
10) loss: 0.15834969376218253
11) loss: 4.620161415685837
12) loss: 3.3183455527703334
13) loss: 0.7204957842954663
14) loss: 0.06191496528422016
15) loss: 0.06191496528422016
finished after 15 iterations at x=[4.99679003 6.00432773 6.99845045 3.00208929 3.00260378 0.99642527
 1.99650952 2.99899164 3.99934955 3.00428565]
Check convergence

---------------------------------------
Trying linear_conjugate_gradient
starting linear conjugate gradient from x0=[4. 7. 8. 8. 4. 6. 3. 5. 4. 8.]
0) loss: 498.3484960448185
10) loss: 0.08033079715158947
finished after 10 iterations at x=[4.36150978 6.85185399 6.69472178 3.42499547 3.51667012 0.29259691
 1.314274   2.80474882 3.87064542 3.85283183]
Check convergence

##############Example-3##############
Trying steepest_descent
starting steepest descent from x0=[8. 7. 6. 3. 5. 1. 7. 8. 8. 7.]
0) loss: 16672.93861321393
100) loss: 20.11378980377664
200) loss: 7.138421371173578
300) loss: 4.961486747822161
400) loss: 3.55575215661532
500) loss: 2.713388225565956
600) loss: 2.2982834892736923
700) loss: 2.019699062216758
800) loss: 1.8511758613787883
900) loss: 1.7278507358028994
999) loss: 1.6222172518963625
finished after 999 iterations at x=[5.14360224 2.58781556 1.21647495 6.0314531  6.35740012 1.72883778
 2.15815335 2.57543314 6.60678996 1.84813268]
Check convergence

---------------------------------------
Trying newton
starting newton method from x0=[4. 4. 6. 4. 6. 2. 4. 8. 8. 3.]
0) loss: 8907.733718516736
1) loss: 1.1430000260132332e-10
finished after 1 iterations at x=[6. 3. 1. 6. 7. 2. 3. 2. 5. 1.]
Check convergence

---------------------------------------
Trying quasi_newton
starting quasi newton from x0=[1. 7. 4. 3. 8. 8. 1. 7. 2. 6.]
0) loss: 563.2088199559109
1) loss: 234.8363346273397
2) loss: 161.65742917374257
3) loss: 134.49197148360946
4) loss: 126.91836276042366
5) loss: 122.84417687720206
6) loss: 115.26547616021467
7) loss: 90.66624055711057
8) loss: 73.14836673558877
9) loss: 7.590954918914492
10) loss: 3.114883723096989
11) loss: 0.006581081069635627
12) loss: 0.006581081069635627
finished after 12 iterations at x=[5.99996984 2.99998177 1.00000361 6.00000367 6.99997767 1.99998725
 2.9999729  2.00001907 5.00006155 1.00002408]
Check convergence

---------------------------------------
Trying linear_conjugate_gradient
starting linear conjugate gradient from x0=[6. 8. 5. 2. 3. 6. 7. 5. 2. 3.]
0) loss: 393.9117677223758
11) loss: 7.573781292489887e-10
finished after 11 iterations at x=[6. 3. 1. 6. 7. 2. 3. 2. 5. 1.]
Check convergence

##############Example-4##############
Trying steepest_descent
starting steepest descent from x0=[2. 4. 7. 7. 4. 6. 2. 1. 5. 7.]
0) loss: 7178.726070829002
100) loss: 9.154581316631269
200) loss: 3.6027397221120707
300) loss: 2.6460855106586507
400) loss: 2.36203743015287
500) loss: 2.193126879177636
600) loss: 2.0619461759417046
700) loss: 1.9394038063523393
800) loss: 1.806333032436139
900) loss: 1.700288303054876
999) loss: 1.584905721339804
finished after 999 iterations at x=[2.34044732 4.21878705 6.10425346 2.60326234 1.74236686 6.20995059
 1.73165977 3.28417992 4.32028281 2.60026384]
Check convergence

---------------------------------------
Trying newton
starting newton method from x0=[2. 2. 4. 1. 3. 2. 4. 6. 2. 1.]
0) loss: 5485.854810328105
1) loss: 2.8047259447902023e-11
finished after 1 iterations at x=[2. 5. 6. 2. 1. 7. 2. 3. 4. 3.]
Check convergence

---------------------------------------
Trying quasi_newton
starting quasi newton from x0=[5. 6. 8. 6. 3. 3. 2. 6. 6. 7.]
0) loss: 449.4869530516138
1) loss: 253.34159419842
2) loss: 218.9937435209738
3) loss: 207.50054287664346
4) loss: 200.9939337697491
5) loss: 181.87998461172748
6) loss: 124.40442517055075
7) loss: 84.4782820608438
8) loss: 28.22226003782471
9) loss: 19.230812338624236
10) loss: 3.6512947755358067
11) loss: 0.0008726541970841666
12) loss: 0.0008726541970841666
finished after 12 iterations at x=[2.00000419 4.99999056 6.00000129 2.00000572 1.00000831 6.99999178
 1.99999677 3.00000255 4.0000034  2.99999569]
Check convergence

---------------------------------------
Trying linear_conjugate_gradient
starting linear conjugate gradient from x0=[1. 2. 1. 8. 1. 6. 4. 4. 3. 6.]
0) loss: 549.1796601165221
11) loss: 8.310403457658498e-08
finished after 11 iterations at x=[2. 5. 6. 2. 1. 7. 2. 3. 4. 3.]
Check convergence

Other methods

Now I will apply every of my 5 predefined functions to every single minimization algorithm I implemented. You will see similar metrics like above, but now additionally you will see the true minima of the predefined function and a plot showing the steps of the algorithm. From this you can clearly see that every method was able to reach every single minima of every single predefined function.

If a predefined function has multiple minima, I tried to hit all of them.

Sometimes a function e.g. steepest descent takes a lot longer but in the end every single method reaches its target in a reasonable bound of error. All the meta data of the methods used and the defined starting points can be found in main.py.

For every single run of a method you can see a plot. The red dot (which is sometimes not visiable any more because my solution is right on top) are the true minimum, the blue star is the start, the green star is my solution and the red line with yellow dots is the path the algorithm took.

In [8]:
main.test_method(m.steepest_descent, 'steepest descent', main.meta_steepest_descent)

starting steepest descent from x0=[-1.5  1.5]
0) loss: 62.50199996800102
10) loss: 0.0075757012910948555
20) loss: 0.002628890558602376
30) loss: 0.0014386089704946927
39) loss: 0.0009758403970935379
finished after 39 iterations at x=[0.94431361 1.05564846]

# MY SOLUTION: 
[0.94431361 1.05564846]
# REAL MINIMIZERS: 
[1 1]
-------------------------------
starting steepest descent from x0=[-1.9  1.5]
0) loss: 9.01273250959996
10) loss: 0.016002977822107488
20) loss: 0.03014062337285495
22) loss: 0.00012554328993871948
finished after 22 iterations at x=[0.67927367 0.67927379]

# MY SOLUTION: 
[0.67927367 0.67927379]
# REAL MINIMIZERS: 
[[ 0.679366 -0.679366]
 [ 0.679366 -0.679366]]
-------------------------------
starting steepest descent from x0=[ 0.  -1.5]
0) loss: 3.583085262730989
7) loss: 0.00046448354870866224
finished after 7 iterations at x=[-0.67917766 -0.67898661]

# MY SOLUTION: 
[-0.67917766 -0.67898661]
# REAL MINIMIZERS: 
[[ 0.679366 -0.679366]
 [ 0.679366 -0.679366]]
-------------------------------
starting steepest descent from x0=[-1.  0.]
0) loss: 7270.6456384560515
10) loss: 0.02918954448540464
20) loss: 0.003934455534135511
30) loss: 0.001852693226028277
40) loss: 0.0011309905558020185
44) loss: 0.0009650082989849819
finished after 44 iterations at x=[1.0554607 2.9445393]

# MY SOLUTION: 
[1.0554607 2.9445393]
# REAL MINIMIZERS: 
[1 3]
-------------------------------
starting steepest descent from x0=[4. 4.]
0) loss: 282.8002828852899
10) loss: 0.028364649477193235
16) loss: 0.0004915069434255112
finished after 16 iterations at x=[2.99999084 2.00001252]

# MY SOLUTION: 
[2.99999084 2.00001252]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting steepest descent from x0=[-3. -1.]
0) loss: 34.9857113690718
10) loss: 0.0003159371082748988
finished after 10 iterations at x=[-3.77931108 -3.28318983]

# MY SOLUTION: 
[-3.77931108 -3.28318983]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting steepest descent from x0=[ 1. -1.]
0) loss: 54.037024344425184
10) loss: 0.5931135039260911
20) loss: 0.002152523344260859
22) loss: 0.0006966726400867778
finished after 22 iterations at x=[ 3.58443111 -1.84810817]

# MY SOLUTION: 
[ 3.58443111 -1.84810817]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting steepest descent from x0=[-1.  4.]
0) loss: 122.70289320142373
6) loss: 4.787620925640046e-05
finished after 6 iterations at x=[-2.80511747  3.13131283]

# MY SOLUTION: 
[-2.80511747  3.13131283]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting steepest descent from x0=[ 1.5 -0.5]
0) loss: 1740.2014251229655
100) loss: 0.7047412437227285
200) loss: 0.484157196097551
300) loss: 0.37625133611735023
400) loss: 0.30616394337637515
500) loss: 0.25856381965227804
600) loss: 0.2217728135639098
700) loss: 0.19329418471290133
800) loss: 0.16225016060095077
900) loss: 0.14341605366103977
1000) loss: 0.12967245021787688
1100) loss: 0.11634222291074812
1200) loss: 0.10738312248032726
1300) loss: 0.09221979746974475
1400) loss: 0.08431052676334025
1500) loss: 0.07509260757431033
1600) loss: 0.06903627375031449
1700) loss: 0.06453465134134684
1800) loss: 0.057921670455245745
1900) loss: 0.05183328480569166
2000) loss: 0.048387416203423385
2100) loss: 0.04401878693230967
2200) loss: 0.039114277790014994
2300) loss: 0.03657096078479617
2400) loss: 0.03357167451870866
2500) loss: 0.030287964785582325
2600) loss: 0.02710980742259755
2700) loss: 0.026141363383582583
2800) loss: 0.023518590138832497
2900) loss: 0.02112337618535591
2999) loss: 0.019695989291512526
finished after 2999 iterations at x=[0.98482787 0.96985602]

# MY SOLUTION: 
[0.98482787 0.96985602]
# REAL MINIMIZERS: 
[1 1]
-------------------------------
In [9]:
main.test_method(m.newton, 'newton', main.meta_newton)

starting newton method from x0=[-1.5  1.5]
0) loss: 62.50199996800102
10) loss: 0.0003259538707011334
11) loss: 9.657892465218728e-05
finished after 11 iterations at x=[0.97109745 1.00578051]

# MY SOLUTION: 
[0.97109745 1.00578051]
# REAL MINIMIZERS: 
[1 1]
-------------------------------
starting newton method from x0=[-1.5  0.5]
0) loss: 3.845126785946076
5) loss: 0.0004840122160097458
finished after 5 iterations at x=[-0.67976391 -0.67962234]

# MY SOLUTION: 
[-0.67976391 -0.67962234]
# REAL MINIMIZERS: 
[[ 0.679366 -0.679366]
 [ 0.679366 -0.679366]]
-------------------------------
starting newton method from x0=[1.  1.5]
0) loss: 3.0468508332374924
4) loss: 0.0009125561115390113
finished after 4 iterations at x=[0.6797482  0.68011222]

# MY SOLUTION: 
[0.6797482  0.68011222]
# REAL MINIMIZERS: 
[[ 0.679366 -0.679366]
 [ 0.679366 -0.679366]]
-------------------------------
starting newton method from x0=[-1.  0.]
0) loss: 7270.6456384560515
10) loss: 0.03791710808557181
14) loss: 0.00029224029519457027
finished after 14 iterations at x=[0.99314903 2.98972354]

# MY SOLUTION: 
[0.99314903 2.98972354]
# REAL MINIMIZERS: 
[1 3]
-------------------------------
starting newton method from x0=[4. 4.]
0) loss: 282.8002828852899
5) loss: 0.00011796258531359892
finished after 5 iterations at x=[2.99999912 2.00000396]

# MY SOLUTION: 
[2.99999912 2.00000396]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting newton method from x0=[-3. -4.]
0) loss: 136.82105101189657
4) loss: 8.618018010543312e-05
finished after 4 iterations at x=[-3.77931067 -3.28318708]

# MY SOLUTION: 
[-3.77931067 -3.28318708]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting newton method from x0=[ 4. -4.]
0) loss: 210.23796041628637
5) loss: 0.00033093856634925484
finished after 5 iterations at x=[ 3.58442938 -1.84813801]

# MY SOLUTION: 
[ 3.58442938 -1.84813801]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting newton method from x0=[-4.   4.5]
0) loss: 228.54430642656578
4) loss: 0.0002034261209230753
finished after 4 iterations at x=[-2.80512054  3.13131417]

# MY SOLUTION: 
[-2.80512054  3.13131417]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting newton method from x0=[-1.5  1.2]
0) loss: 668.823594081429
21) loss: 0.0030125134369809433
finished after 21 iterations at x=[0.99946188 0.99891523]

# MY SOLUTION: 
[0.99946188 0.99891523]
# REAL MINIMIZERS: 
[1 1]
-------------------------------
In [3]:
main.test_method(m.quasi_newton, 'quasi newton', main.meta_quasi_newton)

starting quasi newton from x0=[-1.5  1.5]
0) loss: 9.466339761599441
100) loss: 0.008617747549683483
200) loss: 0.007620815091583316
300) loss: 0.007133215321252921
400) loss: 0.0068207677679096145
500) loss: 0.006595061287860447
600) loss: 0.006420222332107292
700) loss: 0.00627872130707306
800) loss: 0.006160669618156037
900) loss: 0.006059826284129571
1000) loss: 0.005972096000572508
1100) loss: 0.005894649155418648
1200) loss: 0.005825463494314516
1300) loss: 0.005763067599680884
1400) loss: 0.005706346194372284
1500) loss: 0.0056543632984365644
1600) loss: 0.005606467290125221
1700) loss: 0.005562172146866154
1800) loss: 0.00552098912138101
1900) loss: 0.005482512393466324
2000) loss: 0.005446456527302744
2100) loss: 0.005412534656748765
2200) loss: 0.005380528680252251
2300) loss: 0.005350245081921884
2400) loss: 0.00532152091947994
2500) loss: 0.005294240139245897
2600) loss: 0.0052682501546498185
2700) loss: 0.00524345150584547
2800) loss: 0.005219760540077231
2900) loss: 0.005197064701503057
3000) loss: 0.005175525512769553
finished after 3000 iterations at x=[0.90277416 1.09693158]

# MY SOLUTION: 
[0.90277416 1.09693158]
# REAL MINIMIZERS: 
[1 1]
-------------------------------
starting quasi newton from x0=[-1.9 -1.5]
0) loss: 1.2537962200064023
5) loss: 0.0005718685710629894
finished after 5 iterations at x=[-0.67980885 -0.67950393]

# MY SOLUTION: 
[-0.67980885 -0.67950393]
# REAL MINIMIZERS: 
[[ 0.679366 -0.679366]
 [ 0.679366 -0.679366]]
-------------------------------
starting quasi newton from x0=[1.5 1.5]
0) loss: 0.533431698445997
3) loss: 0.00034726533686880686
finished after 3 iterations at x=[0.67962186 0.67962186]

# MY SOLUTION: 
[0.67962186 0.67962186]
# REAL MINIMIZERS: 
[[ 0.679366 -0.679366]
 [ 0.679366 -0.679366]]
-------------------------------
starting quasi newton from x0=[-1.  0.]
0) loss: 1108.9905511946256
100) loss: 0.015128394042035273
200) loss: 0.012632060238859833
300) loss: 0.011490352170125055
400) loss: 0.010785406043700013
500) loss: 0.010289260200036204
600) loss: 0.009913840025768509
700) loss: 0.009613814713021621
800) loss: 0.0093667990713403
900) loss: 0.009158770054433368
1000) loss: 0.008979253300831864
1100) loss: 0.008822122151458731
1200) loss: 0.008682487481630669
1300) loss: 0.00855778764489182
1400) loss: 0.00844506509862048
1500) loss: 0.00834263554723431
1600) loss: 0.00824861000722395
1700) loss: 0.008161786027601522
1800) loss: 0.008081373019872723
1900) loss: 0.00800700794473588
2000) loss: 0.007937185297954575
2100) loss: 0.007871807760720202
2200) loss: 0.007810525753758271
2300) loss: 0.007752667273730167
2400) loss: 0.0076979252288760245
2500) loss: 0.007646091377643451
2600) loss: 0.007596809768832172
2700) loss: 0.007549946003547016
2800) loss: 0.0075053024835559515
2900) loss: 0.007462710075741155
3000) loss: 0.007422345823613678
finished after 3000 iterations at x=[1.10940508 2.89046202]

# MY SOLUTION: 
[1.10940508 2.89046202]
# REAL MINIMIZERS: 
[1 3]
-------------------------------
starting quasi newton from x0=[4. 4.]
0) loss: 30.229601040872463
10) loss: 0.030486128760955043
18) loss: 0.0006588712542057585
finished after 18 iterations at x=[2.99998772 2.00001688]

# MY SOLUTION: 
[2.99998772 2.00001688]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting quasi newton from x0=[-3. -1.]
0) loss: 106.69847840903394
8) loss: 0.0007601172654774584
finished after 8 iterations at x=[-3.77931206 -3.28319517]

# MY SOLUTION: 
[-3.77931206 -3.28319517]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting quasi newton from x0=[ 1. -1.]
0) loss: 124.39981717628551
10) loss: 0.5588177577982238
20) loss: 0.003997957101683604
24) loss: 0.0008822607519621731
finished after 24 iterations at x=[ 3.58443134 -1.84810209]

# MY SOLUTION: 
[ 3.58443134 -1.84810209]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting quasi newton from x0=[-1.  4.]
0) loss: 34.80888872726125
6) loss: 0.00030236536305065196
finished after 6 iterations at x=[-2.80511613  3.13130905]

# MY SOLUTION: 
[-2.80511613  3.13130905]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting quasi newton from x0=[1.5 1. ]
0) loss: 67.48165292278739
100) loss: 0.08684191331679081
200) loss: 0.09167030834709959
300) loss: 0.08906867790014492
400) loss: 0.08443458195466019
500) loss: 0.0804235506896871
600) loss: 0.07414567241424706
700) loss: 0.06443598119745007
800) loss: 0.0565037729923128
900) loss: 0.05016051979876774
1000) loss: 0.01627357350725083
finished after 1000 iterations at x=[1.01615165 1.03261087]

# MY SOLUTION: 
[1.01615165 1.03261087]
# REAL MINIMIZERS: 
[1 1]
-------------------------------
In [12]:
main.test_method(m.non_linear_conjugate_gradient, 'conjugate gradient', main.meta_conjugate_gradient)

starting non-linear conjugate gradient method from x0=[-1.5  1.5]
0) loss: 13.241220902851085
10) loss: 0.004617552485032137
20) loss: 0.0010385169668362198
22) loss: 0.0008672079721478592
finished after 22 iterations at x=[0.95638517 1.05850948]

# MY SOLUTION: 
[0.95638517 1.05850948]
# REAL MINIMIZERS: 
[1 1]
-------------------------------
starting non-linear conjugate gradient method from x0=[-1.9 -1.5]
0) loss: 1.0760356415789971
7) loss: 0.0004751411705734427
finished after 7 iterations at x=[-0.67911176 -0.67897545]

# MY SOLUTION: 
[-0.67911176 -0.67897545]
# REAL MINIMIZERS: 
[[ 0.679366 -0.679366]
 [ 0.679366 -0.679366]]
-------------------------------
starting non-linear conjugate gradient method from x0=[1.5 1.5]
0) loss: 0.18027517566663395
4) loss: 0.00010494366301002801
finished after 4 iterations at x=[0.67928891 0.67928891]

# MY SOLUTION: 
[0.67928891 0.67928891]
# REAL MINIMIZERS: 
[[ 0.679366 -0.679366]
 [ 0.679366 -0.679366]]
-------------------------------
starting non-linear conjugate gradient method from x0=[-1.  0.]
0) loss: 2681.5866661519244
8) loss: 0.0009229938562826045
finished after 8 iterations at x=[1.03872374 2.97044456]

# MY SOLUTION: 
[1.03872374 2.97044456]
# REAL MINIMIZERS: 
[1 3]
-------------------------------
starting non-linear conjugate gradient method from x0=[4. 4.]
0) loss: 111.37575646285444
9) loss: 0.0001880297238504823
finished after 9 iterations at x=[3.00000309 1.99999346]

# MY SOLUTION: 
[3.00000309 1.99999346]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting non-linear conjugate gradient method from x0=[-3. -1.]
0) loss: 65.9336133739851
8) loss: 0.0007612357391164941
finished after 8 iterations at x=[-3.77930334 -3.28317827]

# MY SOLUTION: 
[-3.77930334 -3.28317827]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting non-linear conjugate gradient method from x0=[ 1. -1.]
0) loss: 40.65477430538053
10) loss: 0.012067852557665506
15) loss: 0.00025541320607674623
finished after 15 iterations at x=[ 3.58442832 -1.84811803]

# MY SOLUTION: 
[ 3.58442832 -1.84811803]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting non-linear conjugate gradient method from x0=[-1.  4.]
0) loss: 34.79995404957559
5) loss: 0.0005717990673992933
finished after 5 iterations at x=[-2.80511616  3.1313194 ]

# MY SOLUTION: 
[-2.80511616  3.1313194 ]
# REAL MINIMIZERS: 
[[ 3.       -2.805118 -3.77931   3.584428]
 [ 2.        3.131312 -3.283186 -1.848126]]
-------------------------------
starting non-linear conjugate gradient method from x0=[1.5 1. ]
0) loss: 303.0867772431875
10) loss: 0.01696160009094404
20) loss: 0.01649264624960312
30) loss: 0.0158586815532299
40) loss: 0.015153228846090015
50) loss: 0.014510877034961463
60) loss: 0.01396493153967874
70) loss: 0.01349756876143242
80) loss: 0.01317614774136561
90) loss: 0.0175757130837931
100) loss: 0.014266559960661545
finished after 100 iterations at x=[0.98757111 0.97526649]

# MY SOLUTION: 
[0.98757111 0.97526649]
# REAL MINIMIZERS: 
[1 1]
-------------------------------
In [ ]: